Add deepseek 3.2 exp - #41251
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
Hello, thanks for your support for deepseek v3.2! I wonder when this PR will be ready? |
|
Working on it! Hoping by next week 🤗 |
|
wow this got old! |
|
The submitted code is currently unusable and does not support the Deepseek-v3.2 official version. Is this PR still being updated? |
https://github.com/yunkchen/transformers/tree/v4.57.3_add_dpskv32 |
There is stillDeepseekV32Attention a bug when using LLMC to quantify the model:[rank0]: Traceback (most recent call last): |
New commit pushed, sorry. |
A bug still occurred when running the LLMC quantization model: [rank0]: Traceback (most recent call last): |
I'm seeing problems in this branch with rope factor/beta_fast/beta_slow values not being floats. Is this an oversight? |
Isn't this implementation still O(L^2) since it just masks full attention to the indexer's topk? |
|
@ArthurZucker Is this ready to merge? I'd really love to experiment with some DeepSeek 3.2 Speciale fine tunes. |
|
@ArthurZucker @yunkchen |
|
@ArthurZucker Happy holidays, checking in again :) Can we get this merged please? |
|
Hey! Thanks, just got back from holidays, we shipped https://github.com/huggingface/transformers/blob/57278c904c5158999d31a0db8bfcd63360c37b48 but now I should be able to get back! sorry for the delay everyone v5 needed a slow down in model addition to support all the new features, especially default FP8 weight support! |
|
Thanks @ArthurZucker, do you have an ETA? Getting this in would be massively helpful to me and the community. Happy to help however I can. |
|
Hi @ArthurZucker, just checking in to see if there are any updates on this? There is a lot of interest for this change, so we’re excited to see it move forward! :) Thanks! |
|
Hey all! Yeah I am mega underwater! shipping v5 took / takes longer than we thought, but its still planned. Hopefully for end of week! really sorry its very vague but yeah should be alright |
|
UPDATE: #44054 is gonna take a bit of time so I'll finish this one first |
|
Hi @ArthurZucker, Checking if there is any new updates. Look forward to the DeepSeek3.2 feature! |
…s into add-deepseek-exp
…s into add-deepseek-exp
|
run-slow: auto, deepseek_v3, deepseek_v32, glm4_moe_lite, glm_moe_dsa, longcat_flash, mistral4, youtu |
|
This comment contains models: ["models/auto", "models/deepseek_v3", "models/deepseek_v32", "models/glm4_moe_lite", "models/glm_moe_dsa", "models/longcat_flash", "models/mistral4", "models/youtu"] |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto, deepseek_v3, deepseek_v32, glm4_moe_lite, glm_moe_dsa, longcat_flash, mistral4, youtu |
DeepSeek-V3.2-Exp ships config.json with model_type "deepseek_v32" and no auto_map. On transformers releases without native deepseek_v32 support (our >=4.56 floor still admits these), the AutoConfig lookup that AutoTokenizer performs internally fails and tokenizer loading aborts before any benchmark traffic. --tokenizer-trust-remote-code cannot help because, with no auto_map, there is no remote config class to import. Register a narrow DeepseekV3Config alias under model_type "deepseek_v32" before loading the tokenizer (V3.2 reuses the V3 config schema; vLLM and SGLang do the same). The shim is idempotent and best-effort: a no-op when transformers already knows the model type natively, and silent when the base config class is unavailable so loading reaches its normal error path. Native support landed upstream in huggingface/transformers#41251; this covers the older releases in our supported range that predate it. Fixes ai-dynamo#1047 Signed-off-by: Siwei Li <siweili@amazon.com>
* Add DeepSeek V3.2-Exp (DeepSeek Sparse Attention) - DSA indexer key cache via DynamicIndexedLayer/StaticIndexedLayer (update_indexer) - Add StaticIndexedLayer (static/compile counterpart); remove FP8DynamicIndexedLayer - Register DynamicIndexedLayer + dynamic_sparse_attention causal mask - Isolate glm_moe_dsa indexer diff; keep main's skip-topk feature - Fill in deepseek_v32 docs * Indexer returns the additive index mask directly (drop unused top-k indices) The DSA indexer now builds and returns the [B, S, T] additive sparse mask itself, instead of returning raw top-k indices that were only ever scattered into that mask. GLM-MoE-DSA shares the index mask (not indices) across its skip-topk layers. * Make deepseek_v32 a lean DeepSeek-V3 + DSA model; glm_moe_dsa inherits it - DeepseekV32Attention inherits DeepseekV3Attention (drops the duplicated MLA boilerplate); the indexer returns the additive index mask, folded into the attention mask, no flash padding. - DeepseekV32Model/RMSNorm/RotaryEmbedding inherit DeepSeek-V3; rope_interleave=True. - create_masks_for_generate returns a single mask when all layer_types match (full inheritance, no per-model dict-mask handling). - Indexer scoring uses a plain weighted sum instead of einsum; RMSNorm sourced from DeepseekV3RMSNorm. - Remove unused GatherParallel tensor-parallel style. * Address review: matmul-only indexer, explicit attention forward, importable RoPE - Indexer scoring uses matmul (no einsum) and reuses DeepSeek-V3's apply_rotary_pos_emb_interleave (drops the local single-tensor rope helper). - DeepseekV32Attention writes its MLA forward explicitly (no super().forward), no dead flash padding. - Simplify the indexer key-cache block (in-place k, no else branch); trim verbose comments. - Docs: note the O(index_topk) speedup needs flash_mla (not supported yet). * Revert DynamicCache __iter__ / ddp churn for indexer keys Keep the cache_utils footprint to the essentials (DynamicIndexedLayer, StaticIndexedLayer, update_indexer, dispatch); the indexer key cache is maintained via update_indexer during the forward, so round-tripping it through the legacy/ddp tuple format is unnecessary. * Docs: drop the GlmMoeDsa TIP and contributor line per review * Gate DSA mask materialization to eager/SDPA; indexer returns top-k indices The indexer now returns just the int32 top-k indices. Only the dense eager/SDPA paths materialize the additive [-inf/0] sparse mask from them (so it stays visible to output recorders); a flash-mla kernel would instead consume the indices directly (passed via indices=), avoiding the redundant mask build. GLM threads the top-k indices across shared layers (matching IndexCache), not a prebuilt mask. * Simplify deepseek_v32 checkpoint conversion to the qwen2_moe base The module now uses the checkpoint's own names (indexer wq_b/wk/k_norm/weights_proj, the MLA projections, and gate.e_score_correction_bias), so the old indexer/gate renames are obsolete. deepseek_v32 now uses the plain qwen2_moe expert-fusion mapping, like deepseek_v3 / glm_moe_dsa. * Fix CI: config-attributes check, indexer fp32 dtype + causality - check_config_attributes: allowlist head_dim/layer_types/mlp_bias for the DSA configs; drop the now-unused rope_interleave field (the explicit forward always interleaves). - Indexer: cast the weights_proj input to the layer's own dtype (it is kept in fp32 via _keep_in_fp32_modules but tests may force-cast), fixing fp16/static-cache crashes. - Indexer causality: select top-k only at/before each query — use the passed mask when present (correct for padded/packed inputs), else reconstruct from position_ids (SDPA is_causal skip path). Fixes non-causal SDPA + static-cache. * Make first_k_dense_replace configurable and fix tiny-config MoE test failures - Add first_k_dense_replace config field (default 3) so the dense/MoE layer split is explicit; mlp_layer_types is derived from it in __post_init__. - Tiny DeepseekV32 tester: first_k_dense_replace=1 gives a sparse MoE layer (so test_keep_in_fp32_modules_exist sees e_score_correction_bias) and n_group=1/topk_group=1 keeps group routing valid for 8 routed experts. - Skip test_training_overfit (MoE on a tiny random model is unstable), matching glm_moe_dsa. - Allowlist first_k_dense_replace in check_config_attributes. * more efficient rotary pos emb * manual cleanups * up * updates * nits * nits * nit * nits * remove the overwrite with a post_init fix? * fix tests * long context test * Tidy deepseek_v32 config comments * add padded test and start styling * nit * Apply ruff format to deepseek_v32 / glm_moe_dsa modular files * Add publish/contribute dates to deepseek_v32 model card --------- Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
15 participants
What does this PR do?